Papers with mechanistic understanding of
The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Direction (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel on a variety of reasoning benchmarks, but struggle to generalize to unseen questions due to over-reliance on memorized training examples. |
| Approach: | They propose to identify a set of linear features in the model’s residual stream that govern the balance between genuine reasoning and memory recall. |
| Outcome: | The proposed model can be manipulated to activate the most relevant problem-solving capabilities during answer generation. |
Disentangling Continued Pre-Training: Attention-Driven Routing and Semantic Hub Preservation in Language Adaptation (2026.findings-acl)
Copied to clipboard
| Challenge: | Continued Pre-Training (CPT) enables Large Language Models (LLMs) to acquire second-language capabilities, yet the mechanisms underlying CPT remain poorly understood. |
| Approach: | They investigate how CPT adapts model representations across diverse language families and scripts, model sizes, and architectures. |
| Outcome: | The proposed model can be surgically transferred between base and CPT models with minimal loss. |
How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is a popular choice of safety fine-tuning algorithms, but prior explanations of its effects only account for dampened toxic neurons in the MLP layers. |
| Approach: | They analysed four language models and found that toxic neurons only account for 2.5% to 24% of DPO’s effects across models. |
| Outcome: | The proposed method outperforms DPO in reducing toxicity while preserving perplexity, without requiring any weight updates. |